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ABSTRAC; 

This paper discusses six factors which can be used to 
successfully examine, evaluate, and select reading instructional 
materZals; it also ejcamines the appropriate characteristics of an 
cbje<5t.ive evaluation instrument* Evaluation of reading instructional 
materials should be based on a sound philosophy of reading 
instruction ai^ practiced in the school district; take into 
consideration the intellectual, social, oral, and legal standards of 
amunity and state; utilize all available information regarding the 
Construction and effectiveness of materials being considered; be both 
'^subjective and objective; and deal primarily with, instructional 
desigii characteristics, and only secondarily with physical 
cjiaracteristics* Evaluation instruments shquld be objective, contain 
descriptions to reinforce ratings, utilize inulti-point rating scales, 
and provide mechanisms for comparison with subjective ratings. ^ 
(TS) I . 
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Considerations in Devising Evaluative Factors 

I 

Sel,ection .of reading instructional materials for use in an elementary 

school program is crucial to the success of bliat program. Carefully* 

«. 

selected materials will be appreciated and used constriietively,.,^ 
teachers. - . \ 

There are many fq^ctors whi,ch can bi^ usq4 . to successfully examine, 
evaluate, and select reading instructional materials. Six of these . 
will be discussed here. There will also be discussion of appropriate 
characteristics of an objective evaluation jmstrumajiit and of the role 
of such an instrument in the evaluation-selection process. 

Obviously the final, anjd perhaps the only, determination of the 
effectiveness of materials will come after a period of use. ^^fhil^ 
school districts may find it impractical and expensive to evaluate all 
available ,mateYdals through a year's use, they .should not disregard this 
avenue entirely. To show th^t comparative evaluation is possible, 
/Murray (2^) adopted two linguistic phonic progrqims for use with groups 
of middlegrade sLudents chose^i for similarity in reading difficulty, 
age, 'and in socioeconomic characteristics,' and randomly assigned *to ^ 
treatment. He was able to ^ make conclusions about the overall effe^tive- 
ness of the tvo programs and about their Specific effectiveness when 
used by various subgroups. • ' ' _ 

Althdugh such summ^.^ive evaluation may not be po,ssible as a prelim- 
inary testing of material^* ef f ectiveness ,\ post-adoptipn testing can be 
done. Analysis of the effectiveness of a program after 'a year or so' 
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^ Evaluative Factors 

of use should be a verification of the original selection. If the, 
materials have been chosen ca^fully, they will reflect school and* 
community standards and the program should be effective and should be 
accepte<^ p^ositUvely- 

Most instructional materials select.ion is accomplished by schoc^l 
committee. The committee may act independently or be part of a larger 
district wide committee. It may also be acting only on materials already 
screened by a state selection committee. Whatever the conte>4t, tl^e goal 
9f a reading textbook selection committee is to find and recommend the t 
most effective materials available for teaching reading to a given group/ 
of students. To accomplish this task, the committee must take into 
••account both objective and subjective information regarding the-^student 
population, -to b? served, teaiher styles and preferences-, established * ^ 
school pliilosophy and goals, co;ninunity laws and standards, and publisher s 
Claims for their materials. All of this information must be weighed as, 
evidence against, an objective analysis of the actual content and method- 



ology of materials. The fina 



il 'decision should indicate that the pre- 
fc^-red m^i-terials are of style and quality which- should makq them 
oUCcessXul in the hands oftekchers and students for whom they were 
chcTs-en. The evaluative factors which we will consider here are' intended" 
fo:f considerat.ion, by school peirsbnnel ^s tl}ey charge s.ucli committees .• 
-and by the\ommittees themselJes as theV/^rry out their tasks. 

Evnluatlc^' "oT'reading ins'truction^^ materials sj^ul^^ c based on 
a sound philosoplVy oJ 
, .jfdistrict) > The' first task of the schooT selection , co^nmittee is that _ 
of determining just wh^ is expected of reading instructional materi^. 



)f reading ij:istruction as practiced in the school 
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In schools where teachers believe in ^strong phonics based skill develop*- > 
ment programs materials having these attributes will be appropriate • 
while others might not. To state- a philosophy of reading instruction ' ^ 
requires care and a willingness to ammend as ideas and teachers change. 
However difficult!, such a statement is needed., The whole purpose of 
evaluating materials is to find those which i)est suit the methods and 
likes of teachers and students who will use them.^ Therefore, establish- 
ment of a school philosophy is the essential first step in executing 

such a match. \ ; 11 

Evaluation of reading instructional material^ should take into 
cohsidcration the ' intellectual , social,, moral, and,, legal standards of 
community and state. 'Schools do not operate in vacuums. They are part 
of the lives of communities. Therefore, other aspects of community 
■life' bec,6me important when school decisions are. being made. By taking 
into account the prevailing attitudes and culture of a community ,. an ^ 
evaluation committeeman seek out appropriate materials and simulta- 
neously avoid conflict which might damage school-community relations.. 

This is not' to suggest that schools should' acluiesce to narrow, 
unimaginative, perhaps prejudiced community views . ^ The. school can be 
" an uplifting f^rce' through methods, materials, and 'example. But efforts- 
to lead the community in' new directions will be more likely to succeed 
if they are based<Qj)^wledge of the community's origin^al position. 

Evaluation of reading' instructional materials s hould utilize all \ 

-^j 1 : — 7 \ 

available inforUc^iion regarding - the cons true tipn and effectiveness of 

materials being considered .. Information about materials can and' should 
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, ' . . Page Five 

t ' ^ ' ' Evaluative Factors 

be solicited from whatever sources avaiKible. Publishers base the 
construction of their materials on;a philosophy of Instruction which 
they will communicate to interested school personnel. They conduct 
pilot studies *of materials before they are released for publication. 
They also -conduct field tests. of- the effectiveness of completed materials 



Publishers are a'k^le to provide evidence of materials' ef f,ec tiveness in 
test si tuationsr, whether conducted by their evaluators or by independent 
txxp^rimonters . Other ^schools (districts) which use mater;Lals can pro- 
vide information about effectiveness and subjective inform'ation regard^ 
ing teacher and student' satisf actio^n. The teachers who will actually 
use selected materials will have personal feelings about what' they want 
and what they see in proposed materials. Because they have a^i^important 
interest in materials the teachers^ are the obvious people to .he 1^ 4itake 



objective ratings on items which the evaluation committee has decided 

are important to school philosophy. Representatives of community 

-V • / ^ • • 

interests can offer suggestions, subjective reactions, and perhaps 

some olpjective ratings on"* items they deem essential. , 

t The_contributipns of all o^-^.these people are important to the 

^overall .evaluation^ effoirt. 'To give thenT^aximunf impact, they should 

be considered to be interrelated segments-,, com/lementing and sometimes^" 

overlapping each other. The committee's function is to call^dn each 

% ***** 
of these gj^oups and to coordinate and mak^ itfaximum. use of tl]eir effprts. 

Evaluation has both subjective and objective charactiarisiiics . 

Both typ^s should be utilized, but they should not be confused. 'As 

the committee determines and/or ammends. its 'Statements of schQol phil.o-* 

sophy and community standards, it should simultaneously separate 

objective from subjective items. The objective items can become the 
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basis for an evaluative instrument which can b6 used .to determine ftilt 
what characterizes proposed materials. The subjective items can prj&tide 

a sounding board against which to measure results of objective evali^a- 

i 

N 

tion. ' \ 

The importance of distinguishing objective from subjective infor- 
mation should be emphasised. Appraisal of most available evaluation 
instruments shows a confus^ion of subjectj.ve and objective items, making 

clear decisions about materials difficult if not impossible. 

•»* 

Evaluation should deal '' primarily with instructional design charac- 

« 

teristics , and only secondarily with physical characteristics . Ii) a 
1974 report. (1^) entitled Selecting and Evaluating Beginning Reading 
Materials , the Educational^ Products' Information Exchange (EPIE) suggests 
that reading instructional materials should be evaluated on items 
related to the* following instructional design criteria: goals; c'on.teht - 
(including scope and sequence and subject matter), methodology, and* 
assessment. ' . • . • ' 

The selection committee should certainly concerA itself with , 
these faur areas of instructional design ijii evaluating materials. It 
may want to use them also in establishing its statements of school and 
community philosophy and standards. By so doing the goals, content^ 
methodology,, and assessment aspects of a set of reading materials/can 
be evalucitred -agaijis^ those of school and community. * % 

OnC^'^a^pect of ins true tiona^^ mafcerialis not included here is that 
of physical characteristics. While this omission should not' be taken 
to mean that quality of binding and style of pictures are unimportant, - 
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it does mean that such things are of considerably less importance than 
• are the instructional design aspects discussed above. Physical charac- 

teristics arc obvious and too easily become the focal point of selection 
^ Committees should see that selections ^re made on the basis of the' nj^oi^'e 
essential aspects of .goals, content, methodology, and assessment* 

Evaluation instruments should be objective , contain descriptions 
to reinforce ratings , utilize multi-polAt rating scales , and provide - 
mechanisms for comparison with subjective ratings . . ^^-^ 

An examination of evaluation instrunrents with which most educators ' 
,are' familiar are i'n no way equal to the task. The simplest of them 
amount to little more than yes-no af^sponses to gen^eral and often 
"^trivial items such as Is the prin/t large enough?"' Are pictures appeal- 
^. ing? Is theskill development* prds^ram adequate? After making response^ 
1^/ on instruments of thi5^,kin3"^^^^e^liers. m^ still base decisions on 

idiosyncratic judgement Q,f their own likes and dislikes. In fact, most 
eval/aation instruments serye to cloud vision by requiring ^subjective 
responses. Teachers could probably do equally well to operate on 



^their likes and dislikes from the start. 

One important effort to generate ,ao evaluation instrument which ^ 

.1 

would separate subjective from ob3ective reactions was repor 
O'Brien (3), who asked evaluators to react to the import 




item as Uig-A/ieterrained^whether it was represented jln a given prograp 




Bval^uators checked a criterion ' column if they considered an item 
.important and a judgement coiuinn if they found »the itert)„..present in 



tlie materials.. For each criterion- judgement match, tho^ mateoria 
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awarded a poin^t^^^^ points could be awa-rded at the discretion 

of the evali^^or. > * ' 

The'^ O'Brien instrument represents a step forward by separating 



objective and sub jective - judgements". However, the method of checking 
"Dnly, the presence or abs-ence of an item and only the worthiness or 



.unworthiness of a crite-rion remains a black-white" technique , Both, 

criterion and j'udgemenb*. decisions are. made by' the same person, leaving 

. * *" • 

no opportunity *f or , teacher interaction or for community involvement- 

Since criterion and judgement ratings are made alternately * by the eval- 

uator, inconsistencies might arise from possible confusion caused by 

the ' switching . ^ . 1 

The important concept in the O'Brien instrument is the dual rating 

system. Used by committees^ interacting with 'teras^hers^ and cominuni ty 

representatives, such a system would allow foV ^objective decisions abou\: 

mxiterials' actual value and'lEor subjective decisions about the compati- 

^ bility of the material with school philosophy ^nd goals and with community 

/ standards. If w^<uld allow committees to determine relative, importance 

. J of all aspect^^of - the evaluation and to base final choices on evaluation 

of materials* against the weighted scale. It would allow t(^achers to 

^ take paft in materials' selection while learning a great deal about 



materials being -considered. 



? ^ ... 

One ^^^ajbjrribut^ which should be , added to the instrument is a multi- 
point rating sc^le. A yes/no scale forces decisions based on dichot- 
omies rather than on the sublteties we know to be present in instruc- 
tional materials. One way to .allow forj inclusion o.f sublteties of 
judgement would be . to use five, point rating scales for teachers making 
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* * 

juc3geinents,^abovit what a set, of matferials delivers and fpr committee 

members making -fudgements ^bout-i:he degreq to which a set of ma^rials 

is compatible with school and community standards. Another way would 

be to allow the committee to assign weights to the various items listed 

under instructional design criteria. As inclusion of each item in a ^ 

Set of materials is rated objectively , ^so ife its importance in'^the ' 

subjectiv^-<56ntext of school and community. 

When the evaluation instrument is 'ready, the committee must deter- 
mine how it will be used. Since the instrument was designed with school 
and community g6als and standards in mind, it* inclucjes subjective cri- 
teria against which to examine materials. Therefore, the instrument 
can be used fop: restatement "of school-community priorities ,^ objective 
evaluation of materials* qontents , and objective evaluatic^ of publishers 
claims and^^rcsearch information. While the committee, in the best 
pos'ition ,to restate priori ties and to examine research , 'it^ may want to y 
encourage interaction. with teachers and publishers to conduct objective 
analyses of materials: Pos'sible contributions of teachers and publish- 
ers have already been discussed. 

In summary, the examination and selection of reading instructional 
materials is an essential and coijiplex task. ^ Statements of jschool and 
community standards-_sliauLd- form a basis for evaluation* -Information 



collected from publishers, teachers, community, and research should be 

N, L 

^ considered. Primary emphasis .should b^ placed on essential instructional 
design characteristics. Instruments used in ey^juation should be 

I 

objective aniS allow for comparison with subjective ratipgs. 
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- Attention to the evaluative ^act:ors considered above should insure 
selection of materials appropriate to specific situations, and should, 
promote effective reading instruction* Hovover, it must be noted that* 
tho/best possible materials* selection process wilT not insure* teaclpr ^ 
effectiveness or compensate for poor teaching. Instructional materials 
are only the tangible, inanimate , aids yhich teach'ers employ. The quality 
of utilization of materials will finally decide the question of effective 
reading, ifistructiorf. . " * ^ ' 
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